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ABSTRACT 

The Comprehensive School Reform Demonstration (CSRD) Program 
was established through Public Law 105-78, the 1998 Department of Education 
Appropriations Act. CSRD provides financial assistance to underachieving 
schools implementing comprehensive school-reform programs that are based on 
reliable research and effective practices. It intends to stimulate reform by 
incorporating a comprehensive reform design, support within the school, 
measurable goals for student performance, effective research-based methods 
and strategies, professional development, external technical support and 
assistance, parental and community involvement, coordination of resources, 
and evaluation strategies. This study used case studies of 40 schools and a 
written survey distributed to principals and selected teachers in CSRD 
schools to examine CSRD implementation. Most schools examined served high- 
poverty, highly mobile populations with large concentrations of non-Caucasian 
students. School staff generally supported comprehensive school-reform 
efforts. Most schools were able to coordinate different funding resources to 
support reform efforts. Achievement of success was challenged where high 
principal and teacher turnover made it difficult to maintain the momentum of 
the reform, or changes in external assistance providers slowed progress. 
Further study is needed to determine whether and how these schools can 
sustain their reform momentum. Three appendices contain data-collection 
instruments for different site visits. (Contains 35 exhibits.) (RT) 
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Introduction 



The Comprehensive School Reform Demonstration (CSRD) Program was established through 
Public Law 105-78, the 1998 Department of Education Appropriations Act. CSRD provides 
financial assistance to underachieving schools implementing comprehensive school reform 
programs that are based on reliable research and effective practices. The program intends to 
stimulate comprehensive, schoolwide reform by incorporating nine specific components: 

■ A comprehensive reform design that aligns the school’s curriculum, technology, and 
professional development to enable all students to meet challenging state content and 
performance standards. 

■ Support within the school from teachers, administrators, and other staff. 

■ Measurable goals for student performance and benchmarks for meeting the goals. 

■ Effective research-based methods and strategies for teaching and school management. 

■ Professional development. 

■ External technical support and assistance from an entity with expertise in schoolwide 
reform and improvement. 

■ Parental and community involvement in planning and implementing school 
improvement activities. 

■ Coordination of resources (federal, state, and local) to support and sustain the school 
reform effort. 

■ Evaluation strategies for evaluating the implementation of school reforms and the 
student results achieved. 

The U.S. Department of Education made CSRD funding available under two separate authorities: 
Section 1502 (Demonstration of Innovative Practices) of Title I of the Elementary and Secondary 
Education Act (ESEA) and the Fund for the Improvement of Education (FIE) in Part A of 
Title X of the Elementary and Secondary Education Act. For fiscal year 1998, Congress 
appropriated $120 million under Title I and $25 million under Title X for state CSRD program 
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grants. Funds became available on July 1, 1998. State education agencies received funds 
proportionate to their share of Title I funds and the number of school-aged children they serve. 
These state education agencies awarded subgrants to schools on a competitive basis. Only 
schools eligible to receive funds under Title I, Part A were eligible for section 1502 funds, but 
any school could receive Title X funds. Each funded school was to receive a minimum of 
$50,000 per year for up to three years. 

This study uses two basic methods to examine CSRD implementation in the region served by the 
federal North Central Regional Educational Laboratory (NCREL): in-depth case studies of a 
sample of 40 schools and a written survey distributed to principals and selected teachers in all 
CSRD schools in the region. This report is based on the case studies. 

For the case studies, NCREL staff drew from each state a roughly proportionate sample of 
schools to be involved in a series of three site visits aimed at describing the comprehensive 
school reform implementation process. The study focused on the nine components of school 
reform described in the legislation, the factors that facilitate and inhibit the implementation of 
comprehensive school reform, the types of assistance schools receive from their reform model 
providers and other technical assistance providers, and the ways in which schools integrate their 
comprehensive school reform efforts with other state and local reform efforts. NCREL selected 
schools that proportionally represented urban, suburban, and rural schools, as well as elementary, 
middle, and high schools. 

The evaluation team conducted three rounds of site visits to the 40 schools involved in the case 
study in January and February 2000 (two days), April and May 2000 (one day), and September 
and October 2000 (two days). To the extent possible, the same site visitor conducted all three 
visits to a given school. Exhibit 1 shows the data-collection activities completed during each of 
the three site visits. 
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Exhibit 1 

Data-Collection Activities by Site Visit 



Data Source 


Site Visit 1 


Site Visit 2 


Site Visit 3 


Principal Interview 


X 


X 


X 


Leadership Team Focus Group 




X 


X 


Teacher Interviews or Focus Group 


X 


X 


X 


District StafFInterview 


X 


X 


X 


External Assistance Provider Interview 


X 


X 


X 


Parent Interviews or Focus Group 


X 






Student Focus Group 




X 




Classroom Observation 


X 


X 


X 


Staff Development Form 


X 


X 


X 



Each set of focus group or interview questions revolved around the implementation of the nine 
components of comprehensive school reform. In addition, site visitors collected copies of CSRD 
grant applications, school improvement plans, demographic data, documentation of professional 
development activities, and other background information from each school. The site visitors 
arranged the one- or two-day site visits with each school principal after providing the principals 
with written information concerning the purposes of the study and the types of data-collection 
activities to be included in each site visit. The site visitors left the scheduling of each data- 
collection activity to the discretion of the school principals. Exhibit 2 displays the number of 
schools that received CSRD funds in each of the states in the NCREL region and the number of 
schools that were selected for the series of site visits. 
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Exhibit 2 

Number of CSRD Schools by State 



State 


No. of CSRD 
schools funded 3 


No. of CSRD 
schools sampled 


Percent of CSRD 
schools sampled 


Illinois 


109 


9 


8% 


Indiana 


12 


2 


17% 


Iowa 


10 


2 


20% 


Ohio 


84 


10 


12% 


Michigan 


75 


10 


13% 


Minnesota 


25 


3 


12% 


Wisconsin 


33 


4 


12% 


Total 


348 


40 


11% 



a Data obtained from the Southwest Educational Development Laboratory database at 
www.sedl.org; these data include schools funded as of December 1999. 



Copies of the site-visit data-collection instruments are in Appendixes A, B, and C. 
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Description of the Sample Sites 



NCREL selected the sample sites for this study from the 348 Midwest schools that had received 
CSRD funding as of December 1999. The sample mix represented both the range of grade levels 
served (elementary, middle, and high school) and geographic location (urban, suburban, and 
rural). The study team classified schools serving kindergarten through Grade 8 or kindergarten 
through Grade 6 (or any subset thereof) as elementary schools, schools serving Grades 6 through 
8 or Grades 7 and 8 as middle schools, and schools serving Grades 7 through 12 or Grades 9 
through 12 as high schools. National Center for Educational Statistics (NCES) locale codes of 1 
and 2 (large or mid-size city) were designated as urban, codes of 3 and 4 (urban fringes of large 
or mid-size cities) were designated as suburban, and codes of 5, 6, and 7 (large town, small town, 
and rural) were designated as rural. The study team visited at least one urban, suburban, and rural 
school at each level. Exhibit 3 portrays the sample school distribution by grade level and 
geographic location. The geographic diversity represented by the sites was intended to help the 
study team identify issues and concerns that might be distinct to particular types of schools (e.g., 
urban elementary schools) and might influence the design and implementation of school reform 
efforts. 
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Exhibit 3 

Sample School Sites by Grade Level and Geographic Location 
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Elementary Middle High 

School level 



Note. N~ 40. 



The sample sites varied widely in terms of their demographics. Many of the schools were located 
in densely populated urban neighborhoods with large minority populations and high-poverty 
rates. Often, in both urban and rural settings, the school was the center of activity in the 
community. This section examines various aspects of the sample schools and provides 
information on their similarities and differences. 



Sample School Demographics 

Exhibit 4 illustrates the average school enrollment for each type of site. School enrollment sizes 
varied widely, from 40 students in a rural high school to over 1,000 students in an urban 
elementary school. For the sample sites, the mean enrollment in the urban elementary schools 
was 522, with enrollment figures ranging from 207 to 1,140 students. The mean enrollments in 
the suburban and rural elementary schools were considerably lower at 347 and 359 students, 
respectively. Suburban elementary schools ranged in size from 223 to 426 students, and rural 
elementary schools ranged from 84 to 583 students. The six middle schools ranged in size from 
200 to 730 students, with an average enrollment of 5 1 1 . The two rural high schools were quite 
small, averaging 70 students, and the suburban high school served 630 students. The three urban 
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high schools served an average of 854 students but varied greatly in size: one school served 
1,375 students, whereas the other two each served less than half that number. 



Exhibit 4 

Average Sample School Size 




Elementary Middle High 



School level 

Note. N~ 40 . 

Exhibit 5 illustrates the distribution of minority students in the sample schools. Overall, an 
average of 52 percent of the students were non-Caucasian. However, the demographic makeup 
varied greatly across the sites. Exhibit 5 shows that in 34 percent of the schools, non-Caucasian 
students composed less than one-fourth of the total student enrollment. Yet in 37 percent of the 
schools, non-Caucasian students composed over three-fourths of the total student enrollment. For 
the 18 schools with more than half non-Caucasian enrollment, 16 were predominantly African- 
American and two were predominantly Latino. 



North Central Regional Educational Laboratory 



The Implementation of the CSRD Program — 7 




Exhibit 5 

Minority Enrollment in Sample Schools 



34% 




ED 0% to 25% 

83 26% to 50% 
® 51% to 75% 
□ 76% to 100% 



Other School Characteristics 

On average, 63 percent of the students in the sample schools received free or reduced-price 
lunches (percentages by school ranged from 9% to 99%). The student mobility rate ranged from 
2 percent to 65 percent, averaging 27 percent, and the average daily attendance rate was 90 
percent. The percentage of students with individualized educational plans (IEPs) ranged from 0 
percent to 28 percent, with an average of 12 percent. In 21 of the 37 schools for which we have 
data, no students were classified as English language learners (ELL) or limited English proficient 
(LEP). In the other 16 schools, on average, 8 percent of the students were classified as ELL or 
LEP; the range was from 1 to 61 percent. 

One factor that can directly impact school reform efforts is the amount and type of federal 
funding available to a school, because these funds can be coordinated with CSRD funds to 
enhance school improvement efforts. The study team gathered data regarding the number of 
schools receiving Title I funding and whether the schools used that funding to provide targeted 
assistance or to implement schoolwide projects. Exhibit 6 shows that 19 of the 28 elementary 
schools in the sample implemented Title I schoolwide programs and five had Title I targeted 
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assistance programs. Five of the six middle schools received Title I funding and three of these 
schools implemented Title I schoolwide projects. Only one of the high schools received Title I 
funding; that school implemented a Title I schoolwide project. 

Exhibit 6 

Sample School Sites by Title I Program Status 




Elementary 



Middle 
School level 



High 



on Title I schoolwide ® Title I targeted assistance «No Title I 



Note. N = 40 . 



T\i mover 

When a school implements new instructional models or approaches, several factors can be 
critically important to the success of the implementation, including the administration and 
teaching staff turnover rate. High turnover can slow down or stall the implementation process. In 
the 38 schools with available data, the average teacher turnover rate over the two-year period 
from 1998 to 2000 was 14 percent, ranging from 1 to 33 percent. Almost one-quarter of the 
sample sites had new principals in the 1 999—2000 school year, and 1 5 percent of the schools 
experienced a change in school leadership in 2000-2001 . No differences in staff turnover rates 
were evident across schools by grade level or geographic location. 
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Principal Experience 

Principal experience can be an important factor in the success of school reform efforts. Exhibit 7 
illustrates that 72 percent of the school principals at the 40 sample schools had been in their 
positions for fewer than five years, and 45 percent had held their positions for two years or less. 
Only 16 percent had held their positions for more than six years. Most principals did, however, 
have considerable experience in their districts: half had 16 or more years of experience in their 
districts, and another 25 percent had 6 to 15 years of experience in their districts. 



Exhibit 7 

Principal Experience at Sample Schools 




fHO to 2 years 
® 3 to 4 years 
^ 5 to 6 years 
■ 7 to 8 years 
□ 9 or more years 
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Status of the Nine Components of School Reform 



According to the CSRD legislation, a school receiving program funds must integrate all nine of 
the comprehensive school reform components promoted by CSRD. Schools implementing a 
reform model that does not address all nine components were expected to incorporate the 
missing components into their overall approach to reform. This section summarizes the status of 
each of the nine comprehensive school reform components for which data were available from 
the 40 sample schools (28 elementary schools, 6 middle schools, and 6 high schools). 

Comprehensive Reform Design With Aligned Components 

The CSRD legislation defines this component as follows: 

The program has a comprehensive design for effective school functioning, 
including instruction, assessment, classroom management, professional 
development, parental involvement, and school management that aligns the 
school’s curriculum, technology, and professional development into a schoolwide 
reform plan designed to enable all students — including children from low-income 
families, children with limited English proficiency, and children with 
disabilities — to meet challenging state content and performance standards. The 
program also addresses needs identified through a school needs assessment. 1 

This component encompasses several requirements: 

■ The school must have a written plan. 

■ The plan must be comprehensive. That is, the plan must address instruction, assessment, 
classroom management, professional development, parental involvement, and school 
management. 

■ The plan’s curriculum, technology, and professional development components must be 
aligned with each other. 

’Applications for state grants under the Comprehensive School Reform Demonstration Program, 1998. 
www.ed.gov/offi ces/OESE/compreform/csrdapp.html 
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■ The plan enables all students to meet state content and performance standards. 

■ The plan addresses the needs identified through a school needs assessment. 

In addition to a comprehensive school reform plan, the study team found that 64 percent of the 
schools had a Title I schoolwide project plan; 51 percent had a district-mandated school 
improvement; 31 percent had a state-mandated improvement plan; and 26 percent had some 
other improvement plan. Half of the schools had made minor changes to their original 
comprehensive school reform plans; 37 percent had not made any changes to their original plans; 
and 13 percent had made significant changes to their original plans. There were no evident 
differences in the level of changes made to the comprehensive school reform plans across the 
schools by grade level, geographic location, or reform model type. 

Nearly all of the comprehensive school reform plans (90%) used a school needs assessment to 
identify needs, and two-thirds of the plans addressed how all students (including special 
populations) would be assisted in meeting state content and performance standards. Exhibit 8 
shows the proportion of comprehensive school reform plans that addressed various school 
improvement elements. The most common elements, addressed by at least 80 percent of the 
comprehensive school reform plans, were instructional strategies, professional development, 
assessment, curriculum, and parent involvement. Of the 40 schools, one school addressed only 
one key reform element; 19 schools addressed four to six elements; and 20 schools addressed 
seven to nine reform elements. 
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Exhibit 8 

Percentage of Comprehensive School Reform Plans 
That Addressed Key Reform Elements 



Reform element 


Percent of plans 
including 


Instructional strategies 


95% 


Professional development 


95% 


Assessment 


92% 


Curriculum 


85% 


Parent involvement 


80% 


School management 


67% 


Technology 


59% 


Classroom management 


54% 


Other 


18% 



Note . N = 40. 



The study team asked school staff to estimate how much of their comprehensive school reform 
plans had been implemented thus far. Nearly two-thirds of the schools had implemented most 
(more than 75%) or all elements of their comprehensive school reform plans; only 16 percent 
had implemented fewer than half of their plan elements. No differences in the proportion of 
implemented plan elements were evident across schools by grade level, geographic location, 
model type, or rated strength of principal leadership. 

Exhibit 9 presents the 26 reform models selected by the 40 sample schools organized into three 
categories: curriculum-focused reforms, principle-based reforms, and school improvement 
processes or frameworks. These categories roughly represent a continuum from specific curricula 
as the basis for reform to general school change planning models. 

■ The majority (54%) of the reform models were in the principle-based reforms category. 
These models usually contained some process elements, such as needs assessments and 
comprehensive planning but also featured specific instructional practices that are believed 
to enhance student learning. Schools at all levels implemented principle-based reform 
models. In fact, all of the high schools in the sample implemented this type of model. 
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■ The curriculum-focused reforms category represented 31 percent of the reform models 
and included elementary schools only. 

■ The school improvement processes model category represented 15 percent of the reform 
models and included both elementary and middle schools. 

Four of the 26 reform models implemented by the 40 sample schools were identified as “locally 
developed.” These schools either combined more than one reform model or created their own 
approach by combining specific curriculum, instructional, or organizational practices that 
addressed their needs. 



Exhibit 9 

Summary of School Reform Models 



Model Name 


Model Focus 


Frequency 8 


Curriculum-focused reform models 


Everyday Math 


University of Chicago mathematics curriculum 


1 E 


Early Literacy Learning 
Initiative 

Galaxy Classroom Science 


Language arts and science 


1 E 


Breakthrough to Literacy 


Big books, computer modules 


1 E 


Success for All 


Strategies and structures for teaching reading 


1 E 


Integrated Thematic 
Instruction 


Themes to plan and organize instruction 


1 E 


Reading Renaissance 


Strategies for teaching reading 


1 E 


Locally developed 


Literacy, mathematics, multiage grouping 


1 E 


Principle-based reform models 


Micro Society 


Real-world learning experiences in 6 strands (technology, 
economy, academy, government, arts, heart) 


1 E 


Center for Urban 
Education (DePaul Univ.) 


Instructional planning, professional development, parent 
involvement 


1 E, 1H 


School Achievement 
Structure (DePaul Univ.) 


Sizemore’s 10 routines (e.g., assessment, monitoring, discipline, 
instruction, professional development) 


2 E 


Co-NECT 


Project-based learning, teacher collaboration, technology 


4 E, 1 H 


Expeditionary Learning- 
Outward Bound 


In-depth exploration of topics or themes 


1 E, 1 H 


Modem Red Schoolhouse 


School autonomy, challenging curriculum, technology, individual 
education compacts, parent and community involvement 


1 E 


High Scope 


Key experiences, plan-do-review process, active learning, 


1 E 
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Model Name 


Model Focus 


Frequency 8 




manipulatives 




Effective Schools 


Planning model with seven components (mission, leadership, 
monitoring, high expectations, basic skills, school climate, home- 
school relationships) 


1 E 


Middle Start 


Needs assessment and planning model with focus on instructional 
strategies 


3 M 


Coalition of Essential 
Schools (Sizer) 


10 principles (e.g., focus on learning, mastery, personalized 
learning) 


1 E, 1 H 


Community Learning 
Centers 


Community-based learning, technology 


1 H 


High Schools That Work 


Improving achievement of career-bound students through 10 key 
practices (e.g., high expectations, more challenging academic 
courses, use of assessment data) 


1 H 


Locally developed 


Standards-based curriculum, instructional strategies, professional 
development, collaborative school governance 


1 E 


Locally developed 


Lightspan computer software for home and school use, School 
Improvement Partnership 


1 E 


Locally developed 


Multiage grouping, computers, cooperative learning, project-based 
learning, team teaching 


1 E 


School improvement process or framework reform models 


School Development 
Program (Comer) 


Comprehensive planning process 


2 E, 1 M 


Accelerated Schools 
(Levin) 


Needs assessment, vision, priorities, inquiry process, 
implementing solutions, assessing results 


2 E, 1M 


Standard Bearer for School 
Improvement 


Future orientation, design and focus, strategic action 


1 E 


Turning Points Middle 
School Reform Model 


School climate, technology, teaming 


1 M 



a E = Elementary school, M = Middle school, H = High school. 



About 72 percent of the sample schools were implementing a primary reform model that defined 
their comprehensive school reform approach; 13 percent were implementing multiple reform 
activities that constituted a comprehensive approach to school reform; 10 percent were 
implementing more than one reform model, and 8 percent were implementing multiple reform 
activities that did not constitute a comprehensive school reform approach. Further, 40 percent of 
the schools visited had already begun to implement their model or reform activities prior to 
receiving CSRD funding. No differences in approach to comprehensive school reform were 
evident across schools across by grade level, geographic location, model type, Title I status, or 
rated strength of principal leadership. 
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Support Within the School 

Support from teachers, the principal, and other staff members within the school is essential to the 
maximal effectiveness of a comprehensive school reform effort. Conversely, lack of support 
from one or more entities within a school can hinder reform progress. This section describes the 
level of support by staff members across the 39 sample schools for which data were available. 
Principals and teachers at each school provided percentages of staff members currently involved 
in the reform model implementation effort at their school. Across the sample, principals reported 
that on average 90 percent of the school staff members were implementing the selected 
comprehensive school reform model. Teachers responded similarly. On average, teachers 
believed that 87 percent of staff members were implementing the selected comprehensive school 
reform model. Sixty-four percent of the principals and 47 percent of the teachers reported that all 
teachers were involved in CSR implementation. Twenty-six percent of the principals and 45 
percent of the teachers reported between 50 and 99 percent of the teachers were involved, and 
only 10 percent of principals and 8 percent of teachers reported less than 50-percent teacher 
involvement. In only 3 of the 36 schools with estimates from both principals and teachers were 
the discrepancies between them greater than 20 percent. 

The study team used information from the site-visit interviews to offer possible explanations for 
the low levels of staff support and involvement at certain schools. Based on teacher reports, 
schools were grouped into three categories according to the level of staff involved in reform 
model implementation: (1) schools with less than 80-percent staff involvement (n= 7), (2) schools 
with 80-percent to 90-percent staff involvement («=1 1), and (3) schools with greater than 90- 
percent staff involvement (n= 21). Schools with less than 80-percent staff involvement reported a 
variety of reasons for the resistance of some staff. For example, at one school, problems external 
to the reform plan itself were the primary foci of teachers’ concerns, and teachers paid little 
attention to the reform efforts. A couple of schools lacked staff cohesion — that is, staff members 
were strongly divided among those who supported the comprehensive school reform plan and 
those who felt the plan was unnecessary and a waste of time. In several schools, teachers 
reported that staff other than the school leadership team should be included in reform decisions. 
In one school, staff cohesion was present, but several staff members commented that the 
simultaneous implementation of another program was interfering with their comprehensive 
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school reform implementation effort. Teachers at some schools reported that staff training was 
repetitive and noninstructive or that they were not receiving the guidance or time to implement 
new strategies in the classroom. 

In contrast, several common themes existed among schools whose staff support for the selected 
comprehensive school reform plan was highest. In the majority of these schools, the level of staff 
collaboration in planning instruction was consistently high. Also, in most of these schools, 
teachers were instrumental in the selection or development of the reform plan. Many of these 
schools scheduled weekly or monthly staff meetings to plan activities and implement strategies, 
discuss students’ progress, or discuss topics relevant to instruction or other reform issues. 
Teachers at these schools generally reported having more opportunities to expand their roles, try 
new techniques, and enhance instruction in the classroom. Teachers and students at these schools 
also tended to exhibit a high level of awareness of the selected reform model. 

Staff involvement is critical to reform success. Thus, school staff strove to involve all staff in the 
comprehensive school reform process. Exhibit 10 shows the percentage of schools that used 
various efforts to involve staff. The most popular methods for involving all staff in the 
comprehensive school reform process were providing additional training, promoting committee 
involvement, enlisting the involvement of key school staff, and providing peer coaching or 
assistance. 
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Exhibit 10 

Schools’ Efforts to Involve Staff in Comprehensive School Reform 



Effort 


Percent of 
Schools Using 


Additional training for staff 


54% 


Involvement of staff through committees 


38% 


Enlisting involvement of key staff 


31% 


Peer coaching or assistance 


26% 


Written commitment from staff 


15% 


Use of results to show effectiveness 


15% 


Transfer of nonsupporters to other schools 


13% 


Other 


13% 


Note, n = 39 



Exhibit 1 1 shows which individuals were responsible for the decision to implement the sample 
schools’ selected reform models. In 32 percent of the schools, the principal and school leadership 
team were the primary decisionmakers in the process, and in 29 percent of the schools, the entire 
staff was involved in the decision-making process. No association between the level of staff 
involvement in plan implementation and the persons responsible for the initial decision were 
evident. Additional analyses revealed no differences in staff involvement across the schools by 
geographic location, Title I status, or grade level. 



Exhibit 11 

Parties Responsible for Schools’ Decision to Implement Reform Model 



Primary decisionmaker of model selection 


Percent of 
schools 


The principal and leadership team or other selected group of 
teachers 


32% 


The entire staff 


29% 


Decision made outside the school, e.g. district administration 


21% 


The principal acting alone 


8% 


Other 


10% 


Note, n = 38. 
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Exhibit 12 shows the percentage of sample schools that used various methods of gathering 
information about the reform model prior to selection. Primary methods of information gathering 
across the sample included reading research information related to the model (54%) and 
attending a conference (44%). There were no evident differences in the methods that schools 
used when it came to level of staff involvement. Additional tests revealed no differences across 
schools by geographic location, Title I status, or grade level. 



Exhibit 12 

Methods Schools Used to Gather Information About Reform Models 



Method of gathering information 


Percent of 
schools using 


Reading research information related to this model 


54% 


Attendance at a conference 


44% 


Presentation by model provider 


23% 


Visit to school(s) implementing model 


20% 


District selected model 


15% 


Other 


18% 



Note. N = 39. 



Measurable Goals and Benchmarks 

Comprehensive school reform must involve a clear set of goals and benchmarks to ensure that all 
stakeholders know specifically which goals the school is trying to accomplish and to what degree 
the school has met these goals. Often, reform models require schools to examine and revise their 
existing goals or to develop new goals consistent with the model being implemented. Many 
curriculum-based reform models, such as Breakthrough to Literacy and Success for All, 
introduce new content or teaching methods that require schools to establish new goals. Many 
principle-based reform models — such as Effective Schools, Co-NECT, and Middle Start — place 
high priority on the adoption of clear goals and benchmarks. Much of the initial work of 
implementing these reform models involves clarifying what the school is trying to accomplish 
and specifying the benchmarks the school will use to determine adequate progress. 

The majority of the goals established by CSRD schools involve increasing student achievement 
in reading, language arts, and mathematics. Some CSRD schools’ goals also address student 
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achievement in areas such as science and social studies. The most common methods the schools 
use to assess these areas are standardized tests and statewide assessment programs. Some schools 
also use locally developed instruments, such as portfolios, journals, student exhibitions, and 
rubrics. Or they use commercial assessments, such as the Work Sampling System. Many schools 
have developed additional goals associated with student success in school, including improved 
student attendance, student behavior, and parent involvement. Other goals developed by CSRD 
schools involve intermediate outcomes, such as providing professional development, aligning 
curricula, and acquiring teaching materials and technology. 

Many of the goals established by CSRD schools are expressed in terms of desired student 
achievement scores, attendance levels, or levels of parent involvement in school programs. Some 
schools have identified desired student achievement score levels with descriptors, such as 
developing, competent, proficient, and accomplished. The schools express the benchmarks 
related to these descriptors in various ways. One method is to increase the number or percentage 
of students expected to attain the higher levels each year. For example, a school in which only 30 
percent of the students are at the proficient level might set benchmarks to raise this percentage by 
10 points each year until 90 percent of the students achieve at or above that level. Another 
method is to raise the achievement score level needed to attain each benchmark every year. For 
instance, a designation of proficient may require a test score of 50 the first year, a score of 60 the 
second year, and so on. A third method is to establish several benchmarks and designate the 
number of benchmarks that must be met each year. For instance, one state has established 27 
benchmarks of school effectiveness, and a school must meet at least 25 of these benchmarks to 
be considered effective. 

In general, schools tend to depend heavily upon state assessments to measure progress toward 
meeting their achievement goals. The development of goals and benchmarks and the process of 
aligning them with these external standards is a time-consuming process. Many staff members at 
the sample schools indicated that the time and energy required to perform these tasks were the 
main barriers to the progress of their reform efforts. 

All of the CSRD schools in the site-visit sample had established some level of program goals and 
benchmarks. Nearly 26 percent of the schools had goals and benchmarks in place prior to CSRD 
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funding and had not made any changes since then. Many of these schools had developed goals 
and benchmarks to meet state accountability requirements or as a result of participation in 
activities, such as prior school reform efforts or Title I schoolwide project planning. Staff at 
these schools believed that their existing goals were adequately comprehensive without further 
revision or refinement. Another 31 percent of schools visited reported having revised their 
existing goals and 44 percent reported adding new goals as a result of CSRD funding. No 
differences in the extent to which goals had been changed were evident across the school by 
grade level, geographic location, model type, Title I status, or strength of principal leadership. 

When asked how they had changed their methods of assessing goal attainment since 
implementing comprehensive school reform, 23 percent of the schools indicated that no changes 
had occurred (see Exhibit 13). A majority of the schools, however, indicated using either 
multiple or more frequent assessments, improving the alignment of standards and assessments, or 
making better use of assessment data for planning and decisionmaking. In addition, nearly half of 
the schools reported developing and using rubrics. 



Exhibit 13 

Changes in the Method of Assessing Goal Attainment 



How a school’s method of assessing goal attainment has 
changed as a result of comprehensive school reform 


Percent of 
schools 


Used more frequent assessments 


64% 


Used multiple assessment methods 


62% 


Improved alignment of standards and assessments 


59% 


Made better use of assessment data for planning and 
decisionmaking 


59% 


Developed or used rubrics 


46% 


Added or revised benchmarks 


28% 


No change 


23% 


Other 


3% 


Note. N= 40. 
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Nearly all of the sample schools have aligned their goals with state and district standards. Of the 
schools visited, 78 percent reported having extensively aligned their goals with state standards, 
and 79 percent reported having extensively aligned their goals with district goals (see 
Exhibit 14). One state in the sample school region does not have state standards, but all of the 
schools visited in that state had aligned their goals with district standards. A little more than half 
(52%) of the schools reported having aligned their goals with other standards, such as those set 
by the National Councils of Teachers of Math and English (NCTM and NCTE, respectively). 
Many staff noted, however, that their goals were probably already consistent with those external 
standards. 



Exhibit 14 

Alignment of School Goals With State, District, or Other Standards 



Percent of schools reporting 


Standards with which school’s goals 
and benchmarks are aligned 


Extensive 

alignment 


Some 

alignment 


No 

alignment 


State standards 


78% 


19% 


3% 


District standards 


79% 


16% 


5% 


Other standards (e,g,, NCTM, NCTE) 


26% 


26% 


48% 



Note. n = 37. 



The majority of the sample schools (77%) reported that the principal and school leadership team 
was primarily responsible for monitoring goal attainment (see Exhibit 15). About one-third of the 
schools reported that all staff members were responsible for monitoring goal attainment, and 
schools reported that model providers and district staff were also involved (28% and 41% of the 
schools, respectively). Other groups providing assistance monitoring goal attainment included 
state education agencies, assistant principals, curriculum and assessment coordinators, special 
curriculum teams, and outside evaluators. 
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Exhibit 15 

Parties Responsible for Monitoring School Goal Attainment 



Party responsible for monitoring goal attainment 


Percent of 
schools 3 


Principal and school leadership team 


77% 


District staff 


41% 


Each staff member 


33% 


Model provider 


28% 


Principal (without leadership team) 


15% 


State education agency 


13% 


Other 


13% 



Percentages do not sum to 100 because multiple responses were possible. 
Note, n = 39. 



The study team inquired about the proportion of teacher instruction that was linked to the goals 
and benchmarks of the school reform effort. As Exhibit 16 shows, 55 percent of the sample 
schools reported that the majority (75 to 100%) of instruction was linked to comprehensive 
school reform goals. As a result of spending time and effort developing new goals and reworking 
existing goals and benchmarks, many schools had a greater commitment to increasing student 
achievement and monitoring student progress. The practices of disaggregating test scores and 
using test results to revise instructional and curriculum priorities raised teachers’ awareness of 
student achievement levels. Using alternative assessments of student progress and conducting 
assessment more frequently also resulted in increased overall student achievement. 

Exhibit 16 

Use of Goals and Benchmarks to Guide Instruction 



The extent to which teachers involved with reform efforts Percent of schools 

have used goals and benchmarks to guide instruction reporting 

Most instruction (75% to 100%) linked to goals 55% 

Considerable instruction (50% to 74%) linked to goals 16% 

Some instruction (25% to 49%) linked to goals 24% 

Little instruction (0% to 24%) linked to goals 5% 

Note, n = 38. 
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Effective Research-Based Methods and Strategies 

The U.S. Department of Education describes effective research-based school reform methods 
and strategies as “innovative and proven methods for student learning, teaching, and school 
management that are based on reliable research and effective practices and have been replicated 
successfully in schools with diverse characteristics” (Application for State Grants Under the 
Comprehensive School Reform Demonstration Program, CFDA Number 84.332A). The 40 
sample schools implemented a broad array of strategies in an effort to achieve positive school 
change. To better understand the nature of these changes, the study team asked each school 
whether they had made any of the following types of changes: 

■ Added a new curricular or instructional program (defined as a set of practices and 
procedures that include a specific set of instructional materials). 

■ Implemented new instructional strategies or grouping practices (instructional strategies 
defined as specific actions that promote learning, such as cooperative learning 
techniques, small group instruction, individualized computer instruction, use of 
mathematics manipulatives, and so on; grouping practices included multiage classrooms, 
special education inclusion, looping classes, etc.). 

■ Promoted the adoption of a particular learning theory or pedagogical approach (defined 
as a way of thinking about and organizing curriculum and instruction, such as integrated 
thematic instruction, project-based learning, or multiple intelligences). 

■ Made school governance or school management changes (including block scheduling, 
interdisciplinary teams, reform committees, greater teacher empowerment, etc.). 

Exhibit 17 shows the percentage of sample schools that reported each type of change. Clearly, 
schools could make more than one type of change, and 80 percent of the schools implemented 
new instructional strategies or grouping practices, 74 percent promoted the adoption of a 
particular learning theory or pedagogical approach, 77 percent made school governance or 
school management changes, and 54 percent reported adding a new curricular or instructional 
program. Half of the schools made changes in three of the four areas, and one-quarter of the 
schools made changes in all four areas. No differences in the types of changes made were 
evident across the schools by grade level, geographic location, Title I status, or model type, 
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except that schools using governance models were more likely to report changes in school 
governance or management than schools using other types of models. 



Exhibit 17 

Effective Research-Based Strategies Used 




100% 



0% 



Instructional 

strategies/ 

grouping 



Governance/ 

management 



Theory/ 

approach 



Curriculum 

program 



Type of change 



Note. N = 40. 



New Curricular or Instructional Programs 

Slightly more than half (54%) of the sample schools added a new curricular or instructional 
program. Exhibit 18 shows that the majority of the curricular changes made (13) were in the area 
of reading or literacy and included the adoption of programs such as Accelerated Reader, 4 
Blocks to Literacy, Reading Renaissance, and Success for All. Four of the curricular changes 
made were in the area of mathematics (either implementing Connected Math or Everyday Math), 
and 1 1 of the curricular changes made were in other areas, such as science, technology, or 
careers. 
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Exhibit 18 

Types of Curricular Change 



Curricular changes are: 


Number of schools reporting change 


Elementary 


Middle 


High 


Reading or literacy 


12 




1 


Mathematics 


3 


1 




Other 


5 


2 


4 



Note. N= 40. 



New Instructional Strategies or Grouping Practices 

Eighty percent of the sample schools implemented new instructional strategies or grouping 
practices as part of their school reform effort. Exhibit 19 shows the variety of new instructional 
strategies that schools implemented, including the use of cooperative learning techniques, 
mathematics manipulatives, critical thinking skills, rubrics, questioning strategies, and guided 
reading. The most frequently identified student grouping practices included flexible grouping 
strategies, multiage classrooms, and looping. Of the 28 schools that reported making changes in 
instructional strategies or grouping, 25 percent reported only one change and 75 percent reported 
making multiple changes. 
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Exhibit 19 

Changes in Instructional Strategies or Grouping 



Instruction strategy of grouping practice 


Number of schools reporting 
Elementary Middle High 


Questioning strategies 


2 


1 


Critical thinking or higher-order thinking 


3 


1 1 


Socratic seminars 




1 


Guided reading 


1 


1 


Schoolwide reading time 




1 


Writer workshop 


1 




Mathematics manipulatives 


3 




Integrated science 




1 


Focus on state curriculum standards 


2 


1 1 


Character education or life skills 


1 


2 1 


Cooperative learning 


3 


1 


Flexible grouping 


3 




Multiage classrooms 


2 




Looping 


1 


0 2 


Graphic organizers 


1 


1 


Student learning logs 


1 


1 


Rubrics 


1 


1 1 



Note, n- 28. 



Adoption of a Particular Learning Theory or Pedagogical Approach 

Seventy-five percent of the sample schools reported the adoption of a new learning theory or 
pedagogical approach. Many of these approaches are ways of thinking about or organizing 
curriculum and instruction, but they do not include specific instructional materials. Exhibit 20 
shows that the most commonly identified pedagogical approaches were integrated thematic 
instruction, multidisciplinary instruction, or project-based learning. Other pedagogical 
approaches identified by schools included learning styles, multiple intelligences, constructivism, 
reading across the curriculum, and curriculum mapping. 
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Exhibit 20 

Changes in Learning Theory or Pedagogical Approach 



Pedagogical approach 


Number of schools reporting 


Elementary 


Middle 


High 


Integrated thematic instruction 


3 


3 


2 


Project-based learning 


5 


1 


2 


Reading across the curriculum 




3 


2 


Learning styles 


1 


2 




Multiple intelligences 




2 


1 


Curriculum mapping 


3 






Constructivism 


2 







Note. N = 40. 



School Governance or School Management Changes 

Seventy-seven percent of the sample schools reported school governance or school management 
changes. Exhibit 21 shows that the most frequently identified school government of management 
changes included implementing block scheduling, establishing interdisciplinary planning teams, 
making schedule changes to allow for more teacher planning time, establishing school reform 
committees, and implementing site-based management. Many of these management and 
governance changes aimed to involve teachers in the comprehensive school reform process and 
reflect the changes schools made in their pedagogical approach to school reform. For example, a 
school adopting an integrated thematic instruction model needs time for teachers to meet so they 
can plan interdisciplinary thematic units. Likewise, the use of block scheduling may reflect the 
additional learning time necessary to implement a project-based curriculum. 
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Exhibit 21 

School Governance or Management Changes 



School governance or 
management change 


Number of schools reporting 


Elementary 


Middle 


High 


Block scheduling 


4 


1 


4 


Interdisciplinary teams 


1 


2 


3 


Reform committees 


3 


1 




Schedule changes to increase teacher 
planning time 


5 




1 


Site-based management 


1 






Consensus model 


4 







Note. N= 40. 



Implementation of Classroom-Based Changes 

The study team investigated the extent to which teachers implemented the changes the sample 
schools made as a result of comprehensive school reform. As Exhibit 22 shows, 46 percent of the 
sample schools reported that more than 75 percent of the teachers had implemented new 
curricular or instructional strategies in the classroom. Another 26 percent of the schools reported 
that between 51 percent and 75 percent of teachers had implemented new curricular or 
instructional strategies, and the remaining 28 percent of the schools reported that 50 percent or 
fewer teachers had implemented new curricular or instructional strategies. Those schools that 
reported no teacher implementation of new strategies indicated that the school reform process 
had not yet addressed classroom instruction. There were no differences in the percentage of 
teachers implementing new curriculum or instructional strategies across the three different model 
types. 
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Exhibit 22 

Teacher Implementation of New Curriculum or Instructional Strategies 



46% 




18% 



Percent of 
Teachers 



12376% or more 
EH 51 -75% 

^ 26-50% 

222 1-25% 
®None 



Note. N- 40 . 



Professional Development 

Data provided by 39 of the schools in the site-visit sample indicate that professional development 
activities ranged from many hours of comprehensive, model-focused training to a few hours of 
training on many unrelated topics. The study team used several strategies to collect information 
about professional development efforts related to comprehensive school reform. School staff 
completed a data-collection form by listing staff development dates, the duration of the training 
activities, the number of staff in attendance, the session topics, and the training providers. During 
site visits to the schools, the study team obtained information regarding professional 
development activities through structured and open-ended interview questions posed to 
principals and other staff members. The study team also obtained information through interviews 
with model consultants. 

During the period from June 1998 to September 2000, the sample schools reported spending an 
average of 17 sessions and 170 hours on professional development activities. No differences 
were evident across the schools by grade level, geographic location, or model type for either the 
number of sessions offered or the number of hours spent on professional development. 



30 — The Implementation of the CSRD Program 



North Central Regional Educational Laboratory 



Exhibit 23 shows the percentage of schools by number of professional development activities. 
Thirty-six percent of schools attended between two and 1 0 activities, 25 percent between 1 1 and 
20 activities, and 28 percent between 21 and 30 activities. A small percentage of schools (11%) 
attended more than 30 professional development activities during the comprehensive school 
reform funding period. 

Exhibit 23 

School Staff Participation in Professional Development 



36% 



25% 




28% 



Note, n = 39. 



No. of 
Sessions 



ES2-10 
□ 11-20 
m 21-30 
£2231-40 
S3 4 1-45 



Exhibit 24 depicts the total number of hours of professional development offered between June 
1998 and September 2000. These hours reflect individual, subgroup, and total staff professional 
development activities. About 34 percent of the sample schools had between 45 and 100 hours of 
professional development activities available to staff, and another 34 percent had between 100 
and 200 hours of professional development available. The remaining schools had between 201 
and 445 hours of professional development activities available. 
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Exhibit 24 

Hours School Staff Spent in Professional Development 



34% 




% 



H 45-1 00 
m 101-200 
® 201 -300 
ra 301-400 
■ 400-445 



Exhibit 25 portrays the emphasis the schools placed on various areas of professional 
development. Schools averaged 4.9 areas of major focus and 3.0 areas of minor focus. The 
majority (82%) of the sample schools indicated that developing new instructional skills and 
strategies was a primary focus of the professional development; only 13 percent of the schools 
indicated this area was a minor focus. In 72 percent of the schools, new attitudes or theories 
about learning was a primary focus of the professional development; in only 18 percent of the 
schools was this area a minor focus. Another area emphasized in professional development 
activities was the use of assessment to guide instruction (67% reported this area as a primary 
focus; 23% reported this area as a minor focus). The areas that received the least attention in 
professional development activities were communication skills, classroom management, and 
leadership skills. 

Schools that placed primary focus on developing new instructional skills and strategies were also 
likely to use assessments to guide instruction. Schools that placed primary focus on working with 
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other staff to plan or deliver instruction also were likely to emphasize leadership and 
communication skills. 



Exhibit 25 

The Professional Development Areas Schools Focused On 



Develop new instructional skills and strategies 
New attitudes or theories about learning 
Use of assessment to guide instruction 
Working with other staff to plan or deliver instruction 
Develop and implement new curriculum 
Use of technology 
Leadership skills 
Parent involvement 
Communication skills 
Other educational issues and topics 
Classroom management 

0 % 20 % 40 % 60 % 80 % 100 % 

Note. N = 40 . 




Professional Development Topics 

The study team used the lists of professional development activities that schools provided to 
compute the total number of professional development activities provided in seven general areas. 
Exhibit 26 lists the numbers of sessions devoted to these topics for the 36 sample schools that 
submitted data. Schools participated in an average of 17.2 professional development activities. 
The majority of professional development activities (42%) focused on topics related to 
curriculum and instruction, followed by in-depth training on the selected reform models (16% of 
the professional development activities). Across the sample schools, the fewest professional 
development activities were devoted to assessments and coaching school staff members (7% and 
3% of the professional development activities, respectively). 
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Exhibit 26 

Professional Development Topics 



Professional development topic 


Number of 
activities 


Percent of 
activities 


Number of 
Schools 


Curriculum and instruction 


262 


42% 


33 


Model in-depth 


101 


16% 


25 


Governance and management 


91 


15% 


23 


Model overview 


78 


13% 


21 


Assessment 


40 


7% 


16 


Coaching 


21 


3% 


8 


Other 


25 


4% 


12 


Total 


618 


100% 





Note . n = 36. Data are from June 1998 through September 2000. 



Looking at the number of professional development sessions devoted to each of the seven types 
of activities, one-way analyses of variance (ANOVAs) revealed that schools implementing a 
reform model focused on the overall school improvement process received a greater number of 
in-depth professional development sessions on the reform model itself ( M= 5.33) than schools 
implementing curriculum-focused (M= .89) or principle-based models (M= 2.90), 

F (2,271) = 5.18,/?< .05. Schools implementing principle-based reform models, however, 
received a greater number of professional development sessions focusing on assessments 
( M= 1.76) than schools implementing curriculum-based reform models (M= .33) or overall 
improvement-based models (M = 0.00), F (2,96) = 4.86, p < .05. No differences in the 
professional development topics were evident across the schools by grade level or geographic 
location. 

Barriers to Effective Professional Development 

The study team used data collected during the first two site visits to produce a list of potential 
barriers to effective professional development. Exhibit 27 lists these barriers and the percentages 
of schools indicating these barriers to be a problem. Insufficient time (62%) and a lack of access 
to substitute teachers (67%) were by far the most common barriers to effective professional 
development reported across the sample schools. 
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Exhibit 27 

Barriers to Effective Professional Development 



Barrier 


Percent of 
principals reporting 


Lack of access to substitute teachers 


67% 


Insufficient time for professional development 


62% 


Teacher contract issues (e.g., requirements of additional time or 
pay) 


33% 


Lack of follow-up support and training 


31% 


Lack of access to high-quality trainers 


18% 


Staff turnover 


18% 


Lack of an evaluation system for training and professional 
development 


15% 


Insufficient funds for professional development 


13% 


Content of professional development not matched to staff or 
student needs 


13% 


Lack of staff involvement in the selection of reform models and 
methods 


10% 


Lack of staff involvement in key decisions about professional 
development 


8% 


Other 


20% 


Note . N = 40. 





Of the sample schools, 31 percent cited one of the barriers listed, 33 percent cited two or three of 
the barriers, 23 percent cited four or five of the listed barriers, and 13 percent cited between six 
and ten of the barriers. Subsequent analyses revealed that particular barriers more commonly 
affected some schools than others. These differences were evident across the sample schools by 
model type and grade level; no differences by geographic location were evident. Schools 
implementing a principle-based reform model were more likely than schools implementing 
curriculum-focused or school improvement process reform models to encounter problems 
accessing high-quality trainers, 2 (2, n = 39) = 5.94,/? = .05. In addition, elementary and middle 
schools were more likely than high schools to report having insufficient funds to provide 
effective professional development, 2 (2, n = 39) = 9.01,/? = .01. Elementary and middle schools 
also were more likely than high schools to report a lack of staff involvement in key decisions 
about professional development, (2, n = 39) = 6.66 , p < .05. 
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Efforts to Reduce Barriers to Effective Professional Development 

Some schools reported taking action to eliminate or reduce the barriers to effective professional 
development. During the study team’s fall 2000 site visits, several principals reported that they 
had renegotiated teacher contracts or modified the terms of existing relationships with the reform 
model provider. For example, one principal increased the frequency of the consultant’s site visits 
but decreased the duration of each visit. Several schools sought the assignment of different 
consultants to their schools due to dissatisfaction with the availability of their consultants. Other 
schools used their CSRD funds to hire and train full- or part-time on-site facilitators. Principals 
also reported: acquiring district funds to compensate school staff for excessive overtime hours; 
negotiating with the district for more time and funding for professional development activities 
devoted to reform; requiring all staff to serve on at least one reform committee; and providing 
additional external assistance to help staff learn about using assessment tools to guide 
instruction. 

External Technical Assistance 

CSRD legislation recommends that funded schools receive technical assistance from an external 
source to assist in the implementation of comprehensive school reform. The number of days that 
the schools received on-site technical assistance from an external source (with the source usually 
being the reform model provider) varied greatly. The 31 sample schools that reported these data 
each received between 0 and 80 days of on-site assistance from their reform model provider; the 
median was 6 days. Ten of the 1 1 schools that reported receiving more than ten days of on-site 
technical assistance from their model provider were implementing principle-based reform 
models rather than curriculum-focused or school improvement process reform models. 

Primary Sources of External Technical Assistance 

The study team categorized each of the sample schools according to the primary source of 
technical assistance. Exhibit 28 shows that most (59%) of the 40 sample schools received the 
majority of the on-site technical assistance related to their reform approach from one provider. 
Another 25 percent of the schools received the majority of the technical assistance from more 
than one provider. The majority of the technical assistance that the remaining schools received 
was provided by various external consultants who addressed a range of reform model 
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components (8% of the schools) or by district staff trained by the model developer (5% of the 
schools). Three percent of the schools did not receive any such on-site technical assistance from 
an external source during the 1999-2000 school year. 



Exhibit 28 

Primary Sources of External Technical Assistance 
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Differences in the primary source of external technical assistance across the sample schools were 
evident by geographic location and model type; no differences by grade level were evident. 
Analyses show that schools in rural areas and middle schools were slightly more likely to have 
received the majority of the technical assistance from one on-site provider rather than multiple 
providers, and schools implementing principle-based and school improvement process reform 
models tended to have received the majority of the technical assistance from one provider. 

Overall, sample school staff indicated being satisfied with the primary technical assistance 
providers at their school. Seventy-one percent of the principals were mostly or completely 
satisfied with the technical assistance providers. However, 74 percent of the schools reported that 
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teachers on school leadership teams were mostly or completely satisfied with the technical 
assistance providers, but only 53 percent of the schools reported that teachers not on the school 
leadership teams were mostly or completely satisfied. Schools at which the model provider spent 
a significant amount of time on-site working individually with teachers and providing 
observation feedback tended to express satisfaction with the technical assistance providers. A 
principal of a rural school whose model provider spent a couple of weeks at a time on-site 
providing intensive training and technical assistance expressed strong satisfaction with the model 
provider. This principal reported that the model provider became familiar with the community 
and the teachers and worked hard to specifically address the school’s needs. 

Some schools reported frustration and dissatisfaction with their technical assistance providers, 
because the provider tried to impose a structured model onto the school without addressing the 
school’s individual needs; the model developer did not send the same trainer to the school each 
time, resulting in duplicated training and a lack of trust on the part of the school staff; or the 
model provider did not spend enough time on site. Schools dissatisfied with their reform model 
provider tried to remedy the situation prior to beginning the 2000—01 school year by requesting 
that the model developer assign a different trainer to the school or improve the consistency and 
focus of the technical assistance visits. 

Additional Sources of External Technical Assistance 

Most of the sample schools received additional external technical assistance to supplement the 
assistance provided by the primary technical assistance providers. Exhibit 29 shows that the 
majority (58%) of the schools participated in off-site training or a conference sponsored by the 
model developer. Nearly half (45%) of the schools contracted with the model developer to train 
one or more staff members to serve as an in-house model expert. Thirty-eight percent of the 
schools contracted with external consultants for assistance with specific aspects of 
comprehensive school reform. Typically, these consultants were university professors or college 
instructors with expertise in areas, such as peer coaching or multiage grouping. One-third (33%) 
of the schools received on-site technical assistance from more than one model provider 
representative. Some schools received technical assistance related to specific reform aspects 
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Percent of schools 



from district staff, regional education agencies or laboratories, or other schools implementing the 
same model. 



Exhibit 29 

Sources of Supplemental Technical Assistance 
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Disaggregating these data by geographic location, grade level, and model type revealed these 
findings: 

■ Four of the six rural elementary schools contracted with external consultants who were 
not associated with a model developer or district, and three exchanged information with 
another school that had adopted the same model. 

■ Staff from four of the six suburban elementary schools attended a training or conference 
sponsored by the model developer, and four of the suburban schools received from 
district staff technical assistance on specific areas of the selected reform model. 

■ Ten of the fifteen urban elementary schools sent staff to a training or conference 
sponsored by the model developer, and eight urban elementary schools received technical 
assistance from district staff. 
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■ All three of the urban middle schools sent staff to a training or conference sponsored by 
the model developer, and two urban middle schools received technical assistance from 
external consultants who were not associated with the district or model developer. 

■ The two rural high schools exchanged information with other schools implementing the 
same reform model. 

■ All seven of the schools implementing a school improvement process reform model and 
12 of the 23 schools implementing a principle-based model reported sending staff to off- 
site training or conferences sponsored by the model developer. 

■ Thirteen of the twenty-three schools implementing a principle-based reform model 
contracted with their model provider to train an in-school model expert. 

■ Five of the nine schools implementing a curriculum-focused reform model contracted 
with external consultants to address specific areas of their selected reform model. 

External Assistance front State Educational Agencies 

One-third of the sample schools reported that their state education agency had provided technical 
assistance directly related to their selected reform model. A higher proportion of rural schools (5 
out of 9) reported receiving technical assistance from their state education agency than did 
suburban schools (2 out of 9) or urban schools (6 out of 22). Proportionally, more high schools 
received technical assistance from their state education agency than did schools at lower levels: 4 
out of 6 high schools received such assistance, compared to 7 out of 28 elementary schools and 2 
out of 6 middle schools. 

The sample schools received comprehensive school reform technical assistance from their state 
education agency while planning for reform or after receiving CSRD funding, as Exhibit 30 
shows. Thirty-one percent of the schools reported accessing their state education agency’s Web 
site to obtain or link to information about comprehensive school reform resources. Twenty-eight 
percent of the schools reported having attended state education agency workshops that 
showcased research-based reform models prior to selecting a reform model, and 26 percent 
reported attending workshops on comprehensive school reform issues, such as data-based needs 
assessment, program accountability, and planning for comprehensive school reform. 
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Exhibit 30 

Technical Assistance Provided by State Education Agencies 
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Percentage of CSRD Grants Utilized for External Technical Assistance 

Twenty-six of the sample schools reported the amount of their 1999-2000 CSRD grant funds 
that they spent on external technical assistance (including travel costs for technical assistance 
providers). This amount ranged from 0 percent to 100 percent; the average proportion was 60 
percent. Exhibit 31 shows that six of the schools reported spending 100 percent of their CSRD 
grant on technical assistance, ten of the schools spent between 50 percent and 99 percent of their 
CSRD grant on technical assistance, and ten of the schools spent less than half of their CSRD 
grant on technical assistance. 
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Exhibit 31 

CSRD Grant Funds Schools Utilized for External Technical Assistance 
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Focus for the 2000-01 School Year 

During the fall 2000 site visits, several sample schools — particularly those in the third year of 
their CSRD grant — expressed concern about securing funding to sustain their level of reform 
model implementation. These schools reported taking several measures to sustain their reform 
models, such as arranging for the training of school staff as in-house model experts, purchasing 
as many materials for their curriculum and instructional strategies as possible, organizing action 
teams or other systems of decentralized school governance to ensure staff involvement in 
decisionmaking, and training all staff on the selected reform model. Teachers reported needing 
continual professional development to maintain all of the new teaching strategies and expressed 
concern that with less professional development, they might experience difficulty maintaining 
the reform efforts. Also, many of the schools had high levels of staff turnover and staff at these 
schools expressed concern about how the school would provide new staff with all of the training 
needed to maintain momentum. 
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Parent and Community Involvement 

The study team examined the role of parents and community members in planning and 
implementing comprehensive school reform efforts, general parent awareness of these reform 
efforts, and the activities schools conducted to increase parent and community involvement. The 
study team gathered data regarding the scope of the schools’ previous efforts to involve parents 
and communities to serve as a basis of comparison. Parent focus groups and interviews were the 
primary sources of this information, although at some sites, no parents were available for 
interviews. In these cases, principals or other school staff provided information regarding parent 
and community involvement. 

The study team discovered that increased parent involvement was a key goal in the 
comprehensive reform efforts of about three-quarters of the sample schools. Yet in over one- 
third of the schools, no parents had been involved in the planning process. In addition, 60 percent 
of the schools reported that no community members had been involved in the process. At some 
schools, parents formed committees, investigated possible reform models, served on school 
improvement teams, or served on the team that developed or oversaw the CSRD grant. Many 
schools and districts tried to involve parents and the community by holding public forums on the 
CSRD grant proposal or by conducting parent surveys. 

School administrators at each sample school were queried about the level of parent and 
community participation in the actual implementation of the reform efforts in terms of school 
leadership team membership and opportunities for input. Sixty-five percent of the schools 
reported that at least one parent served on the school leadership team, and 25 percent of the 
schools reported that community organization or business representatives served on the team. 
Eighty-two percent of the schools reported having utilized parent surveys, open meetings, and 
informal discussions to obtain input from parents regarding the implementation of 
comprehensive school reform. Almost half of the schools reported seeking input from the local 
community using similar methods. No significant differences in the number of parents involved 
in comprehensive school reform efforts were evident across the sample school by grade level, 
geographic location, or model type. 
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The study team asked school staff and parents about the level of awareness of reform efforts in 
the general parent population and the community at large. Awareness levels varied widely across 
the sample schools: 38 percent of the schools reported that 75 percent or more of the parents 
were aware of the reform efforts, yet 24 percent of the schools reported that less than one-fourth 
of their parents were aware of the reform efforts. Forty-two percent of the schools estimated that 
most community members were unaware of the reform efforts. 

The sample schools also indicated whether they had a specific plan to increase parent 
involvement. Sixty-eight percent of the schools reported that they did, and almost half of the 
schools also reported that they had a plan to increase community involvement. In many cases, 
however, schools had not immediately begun to implement these plans or take specific steps to 
increase parent and community involvement. In the early stages of the grant period, some 
schools had focused on learning about the selected reform model, giving parent and community 
involvement initiatives less attention. By spring 2000, several schools reported having taken 
specific actions to increase parent and community involvement. 

Exhibit 32 shows that to increase parent involvement, the sample schools most often chose to 
increase the number of parent activities (74%), send home additional information (74%), hire a 
parent liaison or home-school coordinator (44%), or form a parent advisory committee (41%). 
Elementary and high schools (86% and 67%, respectively) were also more likely than middle 
schools (33%) to promote parent involvement by increasing the number of parent activities, 

2 (2, N= 40) = 7.49,/? < .05. Overall, elementary schools ( M= 2.96) and high schools ( M= 2.50) 
used a greater number of methods for increasing parent involvement than did middle schools 
( M= 1 .33), F (2,39) = 4. 1 2, p < .05). No significant differences in the actions taken to increase 
parent involvement were evident across the sample schools by geographic location. 



O 

ERIC 



44 — The Implementation of the CSRD Program 



50 



North Central Regional Educational Laboratory 



Exhibit 32 

Methods Used to Increase Parent Involvement 
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Although the level of community awareness of and involvement in comprehensive school reform 
efforts was lower than that of the school parent population, some schools reported having taken 
specific actions to involve the community. Schools reported providing information to the local 
newspaper and seeking community partners to provide donations or volunteers in the classroom 
or at school events. 

Coordination of Resources 

Authority to manage their CSRD resources afforded many schools greater flexibility in program 
planning and implementation. The vast majority of the sample schools received Title I funds to 
help disadvantaged children meet high standards; these funds served as the other primary means 
of financing the comprehensive school reform efforts. Other funding sources included the Goals 
2000 Program, Title II (for Eisenhower professional development), Title IV (for safe and drug- 
free schools), Title VI (for innovative educational programs), and 21 st Century Learning Grants. 
In addition, schools reported using local curriculum and professional development funds, state 
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general aid or professional development funds, and block grants to support improvement 
activities. 

Exhibit 33 shows the percentage of sample schools that received and used other funding in 
addition to their CSRD monies. Of the schools visited, 68 percent used Title I funds. The 
majority of these schools also operated a Title I schoolwide project which encourages the 
coordination of funding to help students meet high standards. Local and state funds also provided 
an important source of supplemental funding to help schools implement their comprehensive 
school reform programs. 
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Other Funding Sources for Comprehensive School Reform 
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Coordination of resources allows schools to maximize the impact of their CSRD efforts by 
directing as many funds as possible toward those efforts. Many schools have recognized that this 
coordination is especially critical because the CSRD funds will be phased out and the schools 
will need to rely on other funding sources to sustain and maintain their initial reform efforts. 
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Because of the comprehensive scope of most of the selected reform plans, the sample schools 
often used multiple funding sources to finance teacher release time, professional development, 
materials, external technical assistance, and other expenses. One of the advantages of 
establishing a coherent and comprehensive school improvement plan was the school’s ability to 
quickly compare needs with resources and identify the most effective uses of those resources. 

For example, one school was able to respond promptly to a grant application request because its 
improvement plan included a prioritized list of additional assistance (e.g., technology, materials, 
professional development) that would further the implementation of their plan. Principals were 
also positive about their abilities to coordinate resources and indicated that resource coordination 
had enhanced the implementation of their selected comprehensive school reform models. 

Evaluation Strategies 

Most of the sample schools began developing and implementing strategies to evaluate their 
comprehensive school reform efforts after receiving CSRD funding. The schools’ evaluation 
strategies ranged from annually describing their reform model implementation efforts in a report 
to comprehensively evaluating reform model implementation and student assessment data 
utilization. 

Strategies to Evaluate Comprehensive School Reform Implementation 

The 40 sample schools mostly used informal methods to evaluate their efforts to implement new 
curriculum or instructional strategies. The most common evaluation method employed by the 
schools was obtaining teacher input. All of the schools had developed an implementation 
timeline and goals as part of their comprehensive school reform plan, and most schools used 
these timelines to describe their progress implementing the selected reform models in required, 
annual year-end reports to their state education agency or district. Many schools also analyzed 
student grades and test scores, and several schools surveyed teachers, parents, and students. Four 
schools relied on the reform model provider for feedback on program implementation; three 
schools relied on their school improvement team for such feedback. At least five schools 
reported that they had not evaluated their efforts to implement new curriculum or instructional 
strategies. 
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The sample schools evaluated the implementation of new school governance or management 
strategies in much the same manner. Obtaining teacher input was again the most common 
evaluation method. The responsibility for evaluating the implementation of new governance 
strategies or management strategies fell to the school improvement team at five schools and an 
outside evaluator at another; one school reported that an external audit conducted by the district 
had yielded evaluative feedback. Seven schools did not evaluate the implementation of new 
school governance or management strategies. 

Strategies to Evaluate Student Learning 

The 40 sample schools reported relying on a variety of measures to evaluate student learning. 
Exhibit 34 shows that in 1999-2000, 85 percent of the schools used state assessment results, 63 
percent used district-selected standardized test results, 53 percent used student grades, 38 percent 
used student portfolios, and 33 percent employed student assessments recommended by their 
model developer. Furthermore, the majority of the schools (80%) reported using multiple 
measures to assess student learning: 10 percent of the schools used two measures, 40 percent 
used three of the measures, 1 5 percent used four measures, and 1 5 percent used five or six 
measures. 



48 — The Implementation of the CSRD Program 



North Central Regional Educational Laboratory 



Exhibit 34 

Data Schools Utilized to Evaluate Student Learning 
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Utilizing Student Assessment Data to Modify School Reform Efforts 

During the course of this study nearly all of the sample schools showed some change in how they 
used student assessment data to reflect on or modify their comprehensive school reform efforts. 
Some of the schools gained the awareness that they needed to begin critically examining student 
assessment data. Whereas some schools slowly, perhaps reluctantly, developed strategies to 
utilize student assessment data, other schools developed plans and began collecting baseline data 
as soon as they received their CSRD grants. The few schools that had strong evaluation strategies 
in place prior to the study continued to refine how they utilized their student assessment data to 
inform decisionmaking regarding school reform efforts. 

The study team categorized the sample schools into three groups according to the extent to which 
the schools utilized student assessment data to modify school reform efforts (see Exhibit 35). 

The study team based these categorizations on information obtained from a variety of sources 
during the three site visits to each school. The three levels of school assessment data utilization 
are: 
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■ Level 1 — Little or no evidence that student assessment data had been used for program 
improvement. Staff at some Level 1 schools showed little interest in using student 
assessment data to systematically assess student performance, whereas other schools that 
reported feeling unknowledgeable about how to use data for decisionmaking showed 
interest in learning. Some Level 1 schools also reported not yet implementing their plans 
to use student assessment data. 

■ Level 2 — Some evidence that student assessment data had been used for program 
improvement. Level 2 schools had developed strategies to utilize student assessment data 
and were generally in the process of collecting baseline data. In some cases school staff 
were attempting to utilize student assessment data on their own; in other cases the school 
was utilizing some student assessment data but had not adopted a comprehensive system 
for reviewing and interpreting the data. 

■ Level 3 — Strong evidence that student assessment data had been used for program 
improvement. Level 3 schools utilized several strategies to collect and review student 
assessment data and involved all staff in using the data for discussing, planning, and 
modifying their comprehensive school reform efforts. 
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Exhibit 35 

Percent of Schools That Utilized Student Assessment Data 
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Schools That Exhibited Strong Evidence of Utilizing Student Assessment Data 

The study team disaggregated the student assessment utilization data across the sample schools 
by geographic location, grade level, model type, and strength of principal leadership. Analyses 
revealed the following patterns among the three levels of school assessment data utilization: 

■ A higher proportion of rural schools (6 out of 9) exhibited strong evidence that they had 
utilized student assessment data to modify reform efforts than suburban schools (3 out of 
7) and urban (8 out of 20) schools. Urban high schools were fairly evenly distributed at 
each level of student assessment data utilization. 

■ Little or no evidence indicated that middle and high schools had utilized student 
assessment data for program improvement (1 out of 5 middle schools and 1 out of 5 high 
schools), but the majority of elementary schools exhibited strong evidence of utilizing 
student assessment data (15 out of 26 schools). 
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■ Half of the schools implementing a principle-based reform model (1 1 out of 21 schools) 
exhibited strong evidence of utilizing student assessment data for program improvement 
compared to 4 out of 9 schools implementing a curriculum-focused model and 2 out of 6 
schools implementing a school improvement process reform model. 

■ The majority of schools whose principals received a strong leadership rating (1 3 out of 1 8 
schools) exhibited strong evidence of utilizing student assessment data for program 
improvement. In contrast, most of the schools whose principals’ leadership skills 
received a rating of average or poor exhibited little or no evidence of utilizing student 
assessment data for program improvement (5 out of 9 schools and 3 out of 4 schools, 
respectively). 

External CSRD Evaluations 

Many of the sample schools reported that they had participated in several evaluations and 
research studies because of their CSRD grant. In addition to the NCREL study, these studies 
included evaluation site visits or surveys from state education agency staff (53%), the model 
developer (45%), or a CSRD telephone survey contracted by the U.S. Department of Education 
(35%). Ten percent of the schools’ CSRD efforts were evaluated by their districts. 
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Summary and Issues for Further Study 



The Comprehensive School Reform Demonstration program was established in 1998 to provide 
financial assistance to underachieving schools to assist them in implementing research-based 
strategies that show promise for improving student achievement. The present study examines the 
implementation of CSRD in a sample of schools in the region served by the North Central 
Regional Educational Laboratory (NCREL). NCREL staff drew from each state a roughly 
proportionate sample of schools to be involved in a series of three site visits aimed at describing 
the comprehensive school reform implementation process. The sample represented both the 
range of grade levels involved (elementary, middle, and high schools) and geographic locale 
(urban, suburban, and rural). The study focused on the nine components of school reform 
described in the CSRD legislation, the factors facilitating and inhibiting implementation, and the 
types of assistance schools received from external technical assistance providers. 

This report summarizes the findings from three rounds of site visits to 40 schools conducted in 
January and February 2000, April and May 2000, and September and October 2000. The site 
visitors conducted the following data-collection activities at each site: 

■ Interview with the school principal 

■ Focus group or individual interviews with members of the CSR leadership team 

■ Focus groups or individual interviews with a sample of teachers 

■ Observations of classrooms in which teachers were implementing new practices 

■ Focus group or individual interviews with parents 

■ Telephone or in-person interview with district staff involved with CSR 

■ Telephone or in-person interview with the external technical assistance providers 

Each set of focus group or interview questions revolved around the implementation of the nine 
components of comprehensive school reform. In addition, site visitors collected copies of CSRD 
grant applications, school improvement plans, demographic data, documentation of professional 
development activities, and other background information from each school. 
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The major findings include: 



■ Most schools in this study served high-poverty, highly mobile populations with large 
concentrations of non-Caucasian students. The sample included 28 elementary schools, 6 
middle schools, and 6 high schools. The 40 schools included 22 urban, 9 suburban, and 9 
rural schools. A breakout of schools by Title I status revealed that 23 operated Title I 
schoolwide programs, seven had Title I targeted assistance programs, and ten did not 
have Title I programs. School sizes ranged from 40 to 1,375 students. Almost three- 
fourths of the principals had fewer than five years of experience in the school. 

■ The 40 schools selected 26 different school reform models, most of which could be 
characterized as principle-based models that combined school change process features 
with specific instructional practices. The study team classified four of these models as 
locally developed, meaning that the school’s approach to reform was not guided by a 
specific reform model but rather involved a selection of changes in school management, 
teaching strategies, or curriculum. 

■ School staff generally supported comprehensive school reform efforts as a result of 
involvement in reform model selection, training on the reform models, involvement on 
reform committees, or principal support and encouragement for the reform effort. 

■ Nearly all of the schools in the site-visit sample had identified program goals and 
established benchmarks for measuring progress toward those goals. Involvement in 
CSRD caused many schools to reexamine or revise their existing goals and benchmarks 
to make them more rigorous or measurable, and caused over half of the schools to use 
more frequent assessments as well as more assessment methods. Schools generally relied 
upon annual state or district assessments to measure progress toward meeting 
achievement goals. 

■ Eighty percent of the sample schools reported making changes in instructional strategies 
or grouping practices; 77 percent made governance or management changes; 74 percent 
made changes based on a learning theory or approach to instruction; and 54 percent made 
curricular changes. 
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■ The most common professional development activities were designed to develop new 
instructional skills and strategies, to teach new attitudes or theories about learning, to use 
assessment to guide instruction, to help staff work together to plan or deliver instruction, 
and to develop and implement new curriculum. 

■ Schools cited a variety of obstacles to professional development, including lack of access 
to substitute teachers, insufficient time, teacher contract issues involving extra time or 
pay, and lack of follow-up support and training. 

■ The majority of the schools received most of their technical assistance from one external 
provider. Most schools were satisfied with the services they had received from their 
external technical assistance providers. Problems with model providers included not 
enough contact, assistance too general or canned, changes in model providers, or 
insufficiently trained providers. Schools received CSR information from their SEAs via 
mailings, workshops on research-based reform models, workshops on CSR issues, and 
the SEA Web site. 

■ Increased parent involvement was a key goal in the reform efforts of three-fourths of the 
sample schools. Although parents were included on the leadership teams at 65 percent of 
the schools, overall parent awareness of reform efforts was low. Sample schools tried to 
increase parent involvement through offering an increased number of parent activities at 
the school, increasing the amount of communication from the school to the home, hiring 
a parent liaison, or forming a parent advisory committee. 

■ Most schools were able to coordinate resources from different funding sources to support 
reform efforts. Common sources of support included Title I; Goals 2000; 21 st Century 
Learning Centers; and district, state, or federal professional development funds. 

■ Most schools used informal methods, such as staff meetings, to evaluate the 
implementation of new curriculum or instructional strategies. Over 85 percent of the 
schools used state assessments to evaluate student learning; 63 percent used district- 
standardized tests; and 53 percent used student grades. About half of the schools 
consistently used evaluation data for program decisionmaking; one-quarter was 
beginning to use data; and the other quarter had not yet implemented any strategies. 
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Conclusion 



The 40 schools in this sample engaged in reform that was truly comprehensive, covering most, if 
not all, of the nine elements of reform outlined in the CSR legislation. However, it is important 
to note that the schools’ comprehensive reform approaches varied substantially in the nine 
elements they covered and how they covered them. 

Schools selected a broad range of models and approaches that were characterized by the 
evaluation team as curriculum-based, principle-based, or focused on the reform process. Most of 
these schools established satisfactory relationships with external technical assistance providers. 
For the most part, schools did not differ in their approaches to reform across grade levels or 
geographic locale. 

Comprehensive school reform efforts resulted in a number of successes for this group of schools. 
Most schools were able to: 

■ Engage in an increased number of professional development activities 

■ Become more aware of effective, research-based teaching practices 

■ Become more focused on goals and standards 

■ Increase the number and types of assessments they used to measure progress toward 
goals 

■ Begin to evaluate not only the outcomes for students but the implementation of new 
strategies 

However, achievement of these successes was not without challenges. High principal and teacher 
turnover in some schools made it difficult to maintain the momentum of reform and keep all staff 
trained in the new strategies. In some schools, changes in external assistance providers slowed 
progress as well. Finding the time for professional development activities and finding substitutes 
for teachers so that they could participate during their regular workday was problematic for many 
schools. Obtaining parent involvement in the reform process as well as other school activities 
continued to present difficulties. Most schools had little time or expertise to evaluate the 
implementation of their reform strategies or to link implementation to student outcome data. 
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Finally, many schools had serious concerns about the sustainability of their reform efforts once 
CSRD funding ended. These challenges are not unique to the early stages of implementing 
comprehensive school reform but are ones faced by many, if not most, schools engaged in whole 
school reform. 

Overall, CSRD funding presented an important opportunity for schools to initiate comprehensive 
reform. Most of the schools in the sample were in their first year of implementation during the 
1999-2000 school year and showed signs of making progress toward their goals. This study 
attempted to describe the process of reform implementation and found a range of successes even 
among schools implementing the same model. 

Further study is needed to determine whether and how these schools can sustain their reform 
momentum over a longer period of time. The diversity of their approaches argues that in-depth 
descriptive studies, such as this one, will remain important for understanding and explaining 
school’s implementation of comprehensive school reform, their successes and challenges, and 
the assistance they receive from their districts and model providers. Furthermore, such research 
will become more important as schools and other comprehensive school reform stakeholders 
focus increasing attention on the impact of comprehensive school reform on students’ learning. 
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Data Collection Instruments for First Site Visit 
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